PROVIDING PROACTIVE SCAFFOLDING DURING TUTORIAL DIALOGUE USING GUIDANCE FROM 
STUDENT MODEL PREDICTIONS 


Patricia Albacete, Pamela Jordan, Dennis Lusetich, Sandra Katz 
Learning Research and Development Center, University of Pittsburgh, USA 
Irene-Angelica Chounta 
Institute of Education, University of Tartu, Estonia 
Bruce M. McLaren 


Human Computer Interaction Institute, Carnegie Mellon University, USA 


Paper presented at the 19" International Conference on Artificial Intelligence in Education 


London, UK, June 2018 


This research was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant 
R305A150155 to the University of Pittsburgh. The opinions are those of the authors and do not represent views of 
the Institute or the U.S. Department of Education. 


Providing Proactive Scaffolding During Tutorial 
Dialogue Using Guidance from Student Model 
Predictions 


Patricia Albacete), pamela Jordan', Dennis Lusetich!, Irene Angelica Chounta”* 
Sandra Katz!, and Bruce M. McLaren? 


1 Learning Research and Development Center, University of Pittsburgh, Pittsburgh, PA, USA 
? Human Computer Interaction Institute, Carnegie Mellon University, Pittsburgh, PA, USA 
3 Institute of Education, University of Tartu, Tartu, Estonia 
palbacet@pitt.edu 


Abstract. This paper discusses how a dialogue-based tutoring system makes de- 
cisions to proactively scaffold students during conceptual discussions about 
physics. The tutor uses a student model to predict the likelihood that the student 
will answer the next question in a dialogue script correctly. Based on these pre- 
dictions, the tutor will, step by step, choose the granularity at which the next step 
in the dialogue is discussed. The tutor attempts to pursue the discussion at the 
highest possible level, with the goal of helping the student achieve mastery, but 
with the constraint that the questions it asks are within the student’s ability to 
answer when appropriately supported; that is, the tutor aims to stay within its 
estimate of the student’s zone of proximal development for the targeted concepts. 
The scaffolding provided by the tutor is further adapted by adjusting the way the 
questions are expressed. 
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1 Introduction 


Tutorial dialogue systems typically implement a framework called “Knowledge Con- 
struction Dialogues” (KCDs), which guide all students through the same pre-scripted 
“directed line of reasoning” [10], regardless of the student’s ability level. KCDs only 
deviate from the main path in the script to issue “remedial sub-dialogues” when the 
student answers incorrectly, then pop back up to the main path (e.g., [3, 9]). This ap- 
proach can be frustrating and inefficient for some students because they are forced to 
go through long, repetitive and unnecessary discussions due to the dialogues’ lack of 
adaptation to students’ knowledge level. 

One possible way to overcome this limitation is to incorporate a student model in 
the tutorial dialogue system that would emulate how human tutors construct and dy- 
namically update a normative mental representation of students’ grasp of the domain 


content and use this representation to adapt the tutor’s scaffolding to meet students’ 
needs [5]. 

Additionally, a tutorial dialogue system needs policies for how to adaptively struc- 
ture a discussion. Research on human tutoring (e.g., [17]) shows that tutors use their 
assessment of student ability to target scaffolding to the student’s “zone of proximal 
development” (ZPD) [15] —‘“a zone within which a child can accomplish with help 
what he can later accomplish alone” [2]. This work suggests that automated tutors 
should ask challenging questions which the student can answer with adequate support 
and, eventually, be able to answer without assistance. The tutoring system described 
herein implements a decision-making process that attempts to emulate this aspect of 
human tutoring with the support of a student model. 


2 The Adaptive Tutoring System: Rimac 


Rimac is a dialogue-based tutoring system that engages high school students in concep- 
tual discussions after they solve quantitative physics problems (e.g., [1,13]). When us- 
ing the tutor, students typically start by taking an online pretest; they then solve prob- 
lems on paper, such as the one presented in Fig. 1. After working on a problem, students 
use the tutor to watch a video of a sample correct solution and then engage in several 
reflective dialogues, which focus on the concepts associated with the quantitative prob- 
lem. An example of a reflection question is shown in Fig. 1. 


Homework problem statement: Suppose you aim a bow horizontally, directly at the center 
of a target 25m away from you. If the speed of the arrow is 60 m/s, how far from the center 
of the target will it strike the target? That is, find the vertical displacement of the arrow while 
it is in flight. Assume there is no air friction. 


Reflection Question: During the arrow’s flight, how does its horizontal velocity change? 


Fig. 1. A sample homework problem statement and reflection question 


Rimac’s reflective dialogues were designed so that the tutor could provide domain 
and instructional contingency [16] depending on the student’s level of performance. To 
achieve domain contingency—that is, to decide what content to address next—different 
versions of the dialogues were developed, each corresponding to a line of reasoning 
(LOR) at different levels of granularity. These LORs, when embedded in dialogues, 
can be visualized as a directed graph, where nodes are concepts that the tutor queries 
the student about and arrows are the inferences needed to go from one node to the next. 
(See Fig. 2 for a sample segment of a discussion about the reflection question shown in 
Fig. 1. PO > P1 would represent an expert LOR and PO > S1 > S2 > P1 a LOR with 
intermediate reasoning steps.) The system captures such inferences as knowledge com- 
ponents (KCs) which will be used to predict if the student can answer the next question 
correctly. To implement instructional contingency—that is, to vary the way tutor ques- 
tions are expressed—authoring rules were developed to guide how much support to 


embed in a question. For example, node S11 in Fig. 2 can be expressed directly as, “What 
is the value of the acceleration in the horizontal direction?” or with more support as, 
“Given that the horizontal net force on the arrow is zero, what is the horizontal accel- 
eration of the arrow?” (see [12] for a detailed description of these authoring rules). 


PO Pi 


Fig. 2. Graphical representation of the line of reasoning Fnet=0 > v=constant with different 
levels of granularity. Nodes represent questions the tutor could ask. Arcs represent the knowledge 
(KCs) required to make the inference from one node to the next. 


Rimac incorporates a student model which enables it to predict the likelihood of a 
student answering a question correctly [6, 7]. An individual student model is built in 
two steps: first, using the results of the student’s pretest, a clustering algorithm classi- 
fies the student as low, medium, or high. Second, the student is assigned a cluster-spe- 
cific regression equation that is then personalized with the results of the student’s pre- 
test. The regression equation assigned to the student represents an implementation of 
an Instructional Factor Analysis Model (IFM), as proposed by [4]. This student model 
uses logistic regression to predict the probability of a student answering a question cor- 
rectly as a linear function of the student’s proficiency in the relevant KCs. Additionally, 
as the student progresses through the dialogues, his student model is dynamically up- 
dated according to the correctness of his responses to the tutor’s questions. 

Once the tutor engages the student in a reflection dialogue, it needs to decide at what 
level of granularity it will ask the next question in the LOR (or in a remediation if the 
previous question was answered incorrectly), to proactively adapt to the student’s 
changing knowledge level. The tutor will always aim for mastery by selecting the ques- 
tion at the highest possible level that the student can likely answer correctly with ade- 
quate support. In other words, the tutor will choose a question in the highest possible 
LOR that it deems the student will respond correctly or that it perceives to be in the 
student’s ZPD. To make this choice, Rimac consults the student model which predicts 
the likelihood that the student will answer a question correctly. The tutor interprets this 
probability as follows: if the probability of the student responding correctly is higher 
than 60% then the student is likely to be able to respond correctly, and if it is lower than 
40% the student is likely to be unable to respond correctly. However, as the prediction 
gets closer to 50%, there is greater uncertainty since there is a 50% chance that he will 
be able to answer correctly and a 50% chance that he will not be able. This uncertainty 


on the part of the tutor about the student’s ability could be indicative that the student is 
in his ZPD with regards to the relevant knowledge. Hence the tutor perceives the range 
of probabilities between 40% and 60% as a model of the student’s ZPD [6,7]. Thus, the 
tutor will choose to ask the question in the highest possible LOR that has a predicted 
probability of at least 40% of being answered correctly. The expression of the question 
within the LOR is adapted to provide increased support as the certainty of a correct 
answer decreases [12]. 

As an example of how the tutor proactively adapts its scaffolding during a dialogue, 
suppose the student has correctly answered the question at node S1 in Fig. 2. The tutor 
will consult the student model to estimate the likelihood that the student would be able 
to answer P1 (the highest possible node in the LOR) correctly. If this estimate is at or 
above 40% it will ask the corresponding question because this would indicate that the 
student is at or above the tutor’s model of the student’s ZPD for that question. However, 
if the probability is below 40% the tutor will try to pursue the discussion in a simpler 
way and examine S2 in the same manner as with P1. This process is repeated until a 
question can be asked or a leaf is reached in which case the question is asked at the 
highest level of support. As described previously, the tutor further adapts the support it 
provides by adapting how the question at the selected node is expressed. 


3 Discussion and Future Work 


In this paper we described a tutoring system, Rimac, that strives to enhance students’ 
understanding of physics concepts and their ability to reason through “deep reasoning 
questions” [8]. Rimac presents a novel approach to the use of a student model by in- 
corporating the idea of modeling the tutor’s estimate of the student’s zone of proximal 
development which it then uses to guide scaffolding during reflective tutorial dialogues. 
The system takes a proactive approach by anticipating students’ needs and presenting 
a question at each step of a LOR that challenges the learner without overwhelming him, 
and by expressing the question with adequate support. 

The tutor’s scaffolding adheres to the main tenets of contingency, fading, and trans- 
fer of responsibility [14]. This is accomplished by proactively varying the level of com- 
plexity of the knowledge discussed and the way it is expressed and by adapting, step 
by step, to the student’s changing ability. The tutor reduces the support it provides (i.e., 
it fades) as the learner becomes more competent and gradually provides the intermedi- 
ate steps on his or her own (transfer of responsibility). 

An evaluation of the effectiveness of the tutoring system is currently being con- 
ducted. The version of Rimac described herein is being compared with a control ver- 
sion that embeds a poor-man’s student model which assigns students to a fixed LOR 
level in each reflection dialogue based solely on their pretest scores [11]. 
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